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ABSTRACT 



The Project on State-Level Child Outcomes is designed to 
assist states and other groups to improve the measurement of child outcomes 
in state welfare evaluations and in other state data systems. This report 
summarizes a meeting of the 12 states (California, Connecticut, Florida, 
Illinois, Indiana, Iowa, Michigan Minnesota, Ohio, Oregon, Vermont, Virginia) 
that participated in the planning and design phase of the project and 
concerns the development of indicators of children's well-being at the state 
level. Following a copy of the meeting agenda, the report presents summaries 
of the opening remarks by representatives of the funding agencies and the 
technical assistance provider, presentations from representatives from each 
state detailing state efforts to develop indicators for child well-being, and 
remarks from meeting moderators. In addition, the report summarizes 
presentations from experts regarding the use of social indicators for 
evaluation purposes, technical and political issues in developing data 
systems, lessons learned from the Kids Count project, federal data sources 
available to states, opportunities for using telephone surveys for 
state-based estimates of well-being, working with administrative data, the 
importance of developing comparable data sets, and using indicators for 
evaluating welfare reform. The report's appendix includes tables delineating 
child outcome construct areas, intervening mechanisms, and complete reports 
of the presentation on the use of indicators to guide evaluation. The report 
concludes with a list of participants. (KB) 
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Project Overview : The U.S. Department of Health and Human Services 9 Office of Planning, Research, and 
Evaluation at the Administration for Children and Families (ACF), and Office of the Assistant Secretary for 
Planning and Evaluation (ASPE) are working together with states and other groups to improve the measurement of 
child outcomes in state welfare evaluations and in other state data systems. ACF is providing grants to states 
instituting welfare reform demonstrations to augment their demonstration evaluations with measures of child 
outcomes and also to expand their data capability to track state-level indicators of child well-being on an ongoing 
basis. Under funding from ASPE and the other federal contributors and private foundations, the states are receiving 
technical support on these activities from leading researchers who are members of the NICHD Family and Child 
Well-Being Research Network. The Network’s technical support effort is led by Child Trends , Inc. 

The project has two phases. The first phase is a one-year planning and design phase which began October 1, 1996. 
The second phase will be an implementation phase for data collection, analysis, and reporting activities that will 
begin in the fall of 1997. Twelve states participated in the first phase: California, Connecticut, Florida, Illinois, 
Indiana, Iowa, Michigan, Minnesota, Ohio, Oregon, Vermont, and Virginia. 

This report summarizes a meeting on the development of indicators of children’s well-being at the state level. This 
meeting was the third in a series of national-level meetings of the planning phase of the Project on State-Level Child 
Outcomes. 

The project is sponsored by ACF and ASPE. Additional federal funding to support this project has been provided 
by the U.S. Department of Agriculture, the National Institute of Child Health and Human Development (NICHD), 
and the Centers for Disease Control. Several private foundations have contributed funding to support the 
organization of national level meetings, the provision of technical assistance to the states, and the preparation and 
dissemination of written products. These include: the Annie E. Casey Foundation, the Edna McConnell Clark 
Foundation , the George Gund Foundation, and the Smith Richardson Foundation. 

The following individuals have been involved in multiple aspects of this project: 

Administration for Children and Families Office of the Assistant Secretary for 

Howard Rolston Planning and Evaluation 

Alan Yaffe Martha Moorehouse 

Ann Segal 



Child Trends, Inc. 

Christopher Botsko 
Brett Brown 
M. Robin Dion 
Tawanda Greer 
Chisina Kapungu 
Sharon McGroder 
Suzanne Miller 
Kristin Moore 
Erin Oldham 
Martha Zaslow 



NICHD Family and Child Well-Being 
Research Network 
Jeanne Brooks-Gunn 
Natasha Cabrera 
Greg Duncan 
V. Jeffrey Evans 
Kristin Moore 

National Center for Children in Poverty 
Lawrence Aber 
Barbara Blum 
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INDICATORS OF CHILDREN’S WELL-BEING: 

FROM CONSTRUCT TO APPLICATION 
MEETING OF THE PROJECT ON STATE-LEVEL CHILD OUTCOMES 

APRIL 28, 1997 





U.S. Department of Health and Human Services 
Hubert Humphrey Building, Seventh Floor 
200 Independence Avenue, SW 
Washington, DC 20201 


8:30-9:00 


SIGN-IN, CONTINENTAL BREAKFAST 


9:00-9:40 


WELCOME AND OPENING REMARKS 
Howard Rolston, Director, Office of Planning, Research, 
and Evaluation, ACF, U.S. DHHS 
Martha Moorehouse, Senior Policy Analyst, Office of the Assistant 
Secretary for Planning and Evaluation, U.S. DHHS 
Ann Rosewater, Deputy Assistant Secretary, Human Services Policy, U.S. DHHS 
Kristin Moore, President, Child Trends, Inc. 


9:40 


STATE PRESENTATIONS 

Moderators: Martha Moorehouse and Matt Stagner, Office of the Assistant 
Secretary for Planning and Evaluation 


9:40-10:45 


SESSION 1 

Os hi Ruelas, Jill Duerr Berrick; CALIFORNIA 
Stan Sells, Lynn Brant, OHIO 
David Murphey, VERMONT 


10:45-11:00 


BREAK 


11:00-12:00 


SESSION 2 

Judith Erikson, INDIANA 

Jim Murkette, Ron Lucas', CONNECTICUT 

Chuck Johnson, MINNESOTA 

Don Winstead, FLORIDA 

Dave Gruenenf elder, ILLINOIS 


12:00-1:15 


BOX LUNCHES AND BREAKOUT DISCUSSIONS 


1:15 

1:15-2:30 


STATE PRESENTATIONS, CONTINUED 
SESSION 3 

Carol Baron, VIRGINIA 

Marge Reinhart, OREGON 

Deb Bingaman, IOWA 

Susan Wright, Steve Smucker; MICHIGAN 


2:30 


PRESENTATIONS FROM THE EXPERTS 



Moderator: Kristin Moore, Child Trends, Inc. 



2:30-2:45 


SOCIAL INDICATORS AND PUBLIC POLICY IN THE AGE OF 
DEVOLUTION 

Brett Brown, Research Associate, Child Trends, Inc. 


2:45-3:00 


LESSONS LEARNED FROM KIDS COUNT 
Bill O 'Hare, Kids Count Coordinator and Senior Associate, 
The Annie E. Casey Foundation 


3:00-3:15 


SOME FEDERAL DATA RESOURCES 
Christopher Botsko, Research Analyst, Child Trends, Inc. 


3:15-3:30 


BREAK 


3:30-3:45 


SLAITS: NEW OPPORTUNITIES FOR GETTING STATE-BASED 
ESTIMATES IN A FEDERAL SURVEY 

Susan Hauan, Social Science Analyst, Office of the Assistant Secretary for 
Planning and Evaluation, U.S. DHHS 


3:45-4:00 


WORKING WITH ADMINISTRATIVE DATA 
Robert Goerge, Associate Director, Chapin Hall Center for Children, 
University of Chicago 


4:00-4:15 


USING INDICATORS TO EVALUATE WELFARE REFORM: 
RESULTS FROM A DISCUSSION OF STATE WELFARE 
ADMINISTRATORS 

Tom Corbett, Director, Institute for Research on Poverty, 
University of Wisconsin 


4:15-4:30 


SPREADING THE WORD: DISSEMINATION OF INDICATORS 
Linda Kohl, Director, Minnesota Planning 


4:30-4:45 


FORGING AHEAD 

Con Hogan, Secretary, Agency of Human Services, Vermont 


4:45-5:00 


CONCLUDING REMARKS 

Howard Rolston, Director, Office of Planning, Research, and Evaluation, 
ACF, U.S. DHHS 

Ann Segal, Deputy to the Deputy Assistant Secretary for Human Services 
Policy, Office of the Assistant Secretary for Planning and 
Evaluation, U.S. DHHS 



I. 



WELCOME AND OPENING REMARKS 



Dr. Howard Rolston, Director, Office of Planning, Research, and Evaluation, 
Administration for Children and Families, U.S. Department of Health and Human Services 

Dr. Rolston began with an update on state proposals to study the impacts of welfare policies and 
programs. Rolston said that ACF funded nine out of nineteen Track 1 proposals. ACF also 
awarded twenty-four $25,000 planning grants under Track 2 to develop proposals due in July. 

Of the twelve states that are a part of the Project on State-Level Child Outcomes, half received 
Track 1 grants. The remaining states which applied under Track 1 or 2 received planning grants. 
Rolston noted that they received a broad set of proposals and wanted to give everyone a chance 
to develop their proposals further. They would like to provide about $7.5 million in funds to 
support impact studies. For the nine Track 1 proposals, DHHS plains to spend about $5.5 
million. They hope that they can award about $4 million more for Track 2 proposals before the 
end of the fiscal year. Rolston also said that if a state did not receive funding for a Track 1 
project, it should not regard the process as over because there is a substantial amount of money 
available for Track 2 proposals. 

Rolston also noted that they will try to fund all states that submit strong proposals on May 30 
with either or both public and private funds. States were asked to expand their samples of 
children and to budget for in-home surveys because HHS believes that they cam fund these 
studies with available resources. The work that states have done toward developing the common 
core of constructs holds promise that this will be a powerful project. Rolston stated that this has 
been a tremendously successful project so far, and that the work on indicators is as import amt as 
the impact studies. 

Dr. Martha Moorehouse, Senior Policy Analyst, Office of the Assistant Secretary for 
Planning and Evaluation, U.S. Department of Health and Human Services 

Dr. Moorehouse reviewed the project’s framework and the definitions of child impacts and 
indicators, particularly for attendees who were less familiar with the project. Previous meetings 
have focused on measuring child impacts, defined as the differences between children in the 
randomly assigned welfare reform policy group and the AFDC comparison group. Today’s 
meeting focuses on child indicators, defined as measures of children’s characteristics and 
conditions that can be tracked over time or across groups. 

Moorehouse noted a number of reasons that child indicators are important to develop in the 
context of welfare reform. Indicators can make a vital contribution when policy changes occur 
quickly or in multiple areas and a well-designed experiment cannot be fielded in time or with 
adequate controls. In these circumstances, indicators may serve as the only source of 
information on the direction of changes for children, although indicators cannot directly assess 
causal effects. Indicators data also can complement the data from impact studies of welfare 
reform by placing results in the context of broader social and economic trends. For child well- 
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being, indicator data on broader trends in areas such as child poverty, rates of foster case 
placement, or high school completion can be important for interpreting an impact study’s results. 
Using a set of indicators to track child well-being also is a way to look at how children’s needs 
are getting met. Though children have not been highly visible in welfare reform discussions, the 
well-being of children in these families is a serious concern, and significant public resources are 
invested separately in children’s services. Indicators can help identify the additional service 
needs for this population of children. One area in which indicators are becoming more familiar 
is in efforts to develop program performance measures. However, Moorehouse noted that it is 
difficult to causally link indicators data on child well-being directly to program performance. 

ASPE is interested in encouraging states to consider using both administrative and survey data 
sources for indicator work. Each type of data has its strengths and limitations. The work that 
many states are doing already to link administrative data sets is creating a powerful resource for 
producing state-level child indicators and monitoring trends for children over time, and we’d like 
to help states take advantage of this opportunity. 

Moorehouse stressed that impact and indicator approaches are not mutually exclusive. These 
approaches can be used to good advantage in combination. Going beyond today’s meeting, this 
is an important time for states to build a portfolio of research and evaluation initiatives using 
different methods and data sources to provide short- and long-term information on children’s 
well-being at the state and local level. States have new needs and responsibilities for monitoring 
children’s well being and also for assessing how children are faring in relation to state 
innovations in policies and services. As an example, a state’s portfolio might include some or all 
of the following: an augmentation of a national survey sample to obtain comparable state-level 
data on child indicators now available only nationally, other child indicators developed from 
administrative data, child indicators developed from a state-based survey assessing areas of 
special priority, and also experimental or non-experimental evaluations of specific programs 
using a core of common child measures which provide crosswalks with other data on children’s 
well being. 

The goal for today’s meeting is to discuss productive uses of indicator data, to focus on how to 
report data, not just on the best way to collect or link it, and to examine some of the difficult 
technical issues, especially in the quality of administrative data. The credibility and integrity of 
the data are paramount. Working toward a common core of indicators that have comparability 
across states is also important. However, today’s starting point is for each state to focus on 
indicators of interest and work on measuring these indicators well. 

Ann Rosewater, Deputy Assistant Secretary, Human Services Policy, U.S. Department of 
Health and Human Services 

Ms. Rosewater acknowledged the hard work of all of the project participants and said that this 
project has been a productive collaboration across states, the federal government and other 
partners. For many years, the federal government has been monitoring and tracking the well- 
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being of children over time. The Office of the Assistant Secretary for Planning and Evaluation 
(ASPE) has used indicators to monitor the outcomes of existing policies and the need for new 
policy efforts. At the present time, ASPE is sponsoring three major indicators projects. Each 
project will produce an annual report that will address the well-being of children at the national 
level. Each report will also vary in the scope of child well-being that it covers, the style of 
presentation, and the audience for the report. Rosewater expressed her hope that in each case, the 
reports that will be produced will help states consider how to move forward with indicators that 
are consistent, credible, and have integrity. 

The first report will be the 1997 edition of Trends in the Well-being of America 's Children and 
Youth, a background source document for those conducting children’s policy work. This report, 
first published in May 1996, will include about 70 indicators. These are produced regularly by 
federal statistical agencies, although not on the same schedule. The production of this report 
represents the first time that the federal government agencies have brought together a select 
group of indicators. The 1996 report is available on ASPE’s web site, as will be the new edition. 

The second report that will be released is America ’s Children: Key Indicators of Well-being. 
Rosewater described an executive order that the President issued in the past week which 
established a Federal interagency forum on child and family statistics. This report, designed to 
appeal to a broader audience, will present 25 indicators selected by the interagency forum. In 
selecting the set of key indicators, the forum consulted with a broad group of people to determine 
which indicators would be most useful to describe the well-being of children. Rosewater said 
that these key indicators can be tracked over time, and she hoped that this report can be used as a 
model by states. She also asked states to push the Federal government to develop additional 
indicators. This report will be released to the public at the end of July. 

The third project that Rosewater described is a requirement of a 1994 statute initiated by Senator 
Daniel Moynihan. The project is studying which statistics, if collected on a regular basis, could 
be most useful in tracking dependence on means tested cash assistance and non-cash assistance 
programs. The project produced an interim volume with 60 indicators. ASPE is required to 
produce an annual volume which will include a set of indicators that will focus on welfare 
dependence and the well-being of children and adults. Finally, Rosewater welcomed input on the 
selection of indicators for these projects. They are eager for input on which indicators do not 
provide enough information and which ones should perhaps be dropped. 

Dr. Kristin Moore, President, Child Trends, Inc. 

Dr. Moore noted that the project participants have been making excellent progress in designing 
their impact studies. These studies will produce important information on how differing 
approaches will affect children and families. While proceeding with their impact studies, state 
representatives have been asking what more they will get by collecting indicators data. Moore 
noted that there are several ways in which indicators data can be useful. First, indicators data, 
unlike impact studies which capture outcomes at a point in time, can present a picture of how 
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children are faring over time . In addition, information can be collected on aU children in a state. 
States can also track trends in entry into the welfare system. For example, if new policies 
discourage entry into the welfare system, states can determine through indicators data if this is 
actually occurring. 

Moore used the example of an imaginary state called "Inyminnegandia" to illustrate how 
indicators data can be used. She presented three hypothetical scenarios from 1996 to 2005 for 
four indicators: the welfare caseload; the nonmarital birth rate; the poverty rate for all children; 
and the foster care caseload. In the first scenario (see Scenario 1 in Appendix, page 24), if states 
are only tracking welfare caseloads, one can see that the welfare caseloads have decreased. 
However, state policy makers tracking a broader array of child and family indicators will see that 
nonmarital birth rates have stayed high, poverty rates have not changed, and the foster care 
caseload has stayed about the same. 

These additional data suggest a less optimistic picture then one based solely on caseload data. In 
the second scenario (see Scenario 2 in Appendix), the welfare caseload has declined, although 
not as dramatically as in Figure 1 . The nonmarital birth rate initially has risen somewhat 
dramatically, and then declined after five or six years. The child poverty rate has increased but 
then declined, and the foster care caseload has risen slightly before it slightly declined. In this 
scenario, the magnitude of improvement varies across indicators. In the third scenario (see 
Scenario 3 in Appendix), the welfare caseload has declined substantially and continuously. 

There was a slight upward blip in the nonmarital birth rate and in the foster care caseload, and 
then they declined considerably. Finally, the child poverty rate declined gradually at first, and 
then substantially. 

In all three scenarios, the welfare caseload declined over time. However, the other indicators 
varied in their trends over time, and these suggest very different forecasts for child well-being. 
Moore concluded that, as wonderful as impact data are for studying the implications of welfare 
reform, indicators data can significantly enhance our understanding of children in a particular 
state over time. 

II. STATE PRESENTATIONS 

Moderators: Dr. Matt Stagner and Dr. Martha Moorehouse, Office of the Assistant 
Secretary for Planning and Evaluation, U.S. Department of Health and Human Services 

The twelve participant states selected a representative or representatives who described, in a brief 
presentation, the indicators efforts that are underway or are planned in their state. 
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Session 1 State Presentations 



1. Oshi Ruelas, California 

Ms. Ruelas first provided an update on California’s child impact study plans. She said that they 
have been working with various agencies and have found several data bases with information 
about child well-being that they can use. They have learned that WIC collects information on 
language development and takes measurements of children’s heads, arms and legs. In addition, 
the Preventive Health Care agency has data on services that pregnant women receive, as well as 
complications during their pregnancies and birth outcomes (low birth weight, developmental 
disabilities, APGAR scores and services that children receive). However, the data are available 
only for those children bom at risk. The Child Health and Disability Program collects data on 
services children with disabilities receive in their first two years of life. By the end of the month, 
California will have information from several different agencies about data that they collect on a 
regular basis. 

With regard to California’s efforts to link administrative data, the University of California at 
Berkeley has been creating the California Children’s Services Archive. Their goal is to link 
AFDC, foster care, juvenile justice, and education data. Some data, such as live birth records, 
incarcerations and parole, foster care placement information, adoptions, death records, special 
education disabilities and placement information, are already in the archive. UC Berkeley plans 
to add data on child abuse and neglect, probation, neighborhood and community indicators, 
employment, education, mental health data and medical records over the next five years. Their 
long-term goal is to add education data from the local school districts. 

2. Stan Sells and Lynn Brant, Ohio 

Mr. Sells described a proposal that Ohio has submitted to the state legislature to target three 
domains: prevention, self-sufficiency attainment and labor market retention. They are proposing 
to prevent and divert families from going on public assistance and to track indicators that would 
measure these efforts. Proposed indicators include: the out-of-wedlock pregnancy rate, the 
paternity establishment rate, child support collections and diversions. For those already on 
assistance, Ohio plans to track work activity participation, the job placement rate, TANF benefit 
costs, educational attainment, duration of services, the percentage of child care requests and the 
quality of services. They also plan to track several indicators of labor market retention. Ohio 
also plans to track several indicators related to child protection including child abuse and neglect 
reports, incidence and duration of out-of-home placements, and length of time that children wait 
until they are adopted. 

Sells also described the governor’s Family and Children First Initiative. This initiative was 
designed to assure that children are healthier by the year 2000 as measured by the number of low 
income women receiving prenatal care, the number of drug free births, immunizations, and the 
percent of eligible children in Head Start and other preschool programs. An additional goal of 
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the initiative is to improve family stability by reducing the number of children experiencing 
multiple out-of-home placements. Ohio also has several programs which are targeted toward 
reducing the teen pregnancy rate. The state’s LEAP program has collected data since the late 
1 980's. The state’s LearnFare project tracks TANF families and excessive absenteeism from 
school. LEAP is state wide but LearnFare is in a limited number of counties. Through these 
initiatives, Ohio will have data on a number of participants. 

Mr. Brant discussed Ohio’s Data Warehouse Project. He noted that human services agencies 
are being asked questions now that they had not even thought about ten years ago. They are 
often in the position of not having data or not having access to data to answer these questions. 

Or, when data exists, it is not collected in a way that lends itself to straightforward analyses. In 
addition, there is a lack of linkages across data systems. For example, it would be difficult to 
determine whether parents of a child in the foster care system are receiving TANF. Thus, Ohio 
has embarked on a data warehouse project to address these problems. They intend for it to serve 
as an overarching structure to link systems and to "drill down to the most granular level" within 
each data system. This will allow Ohio to look at families and children in a holistic way. Once 
it is completed, the Ohio Data Warehouse will be one of the largest in the country. They do not 
yet know how many direct users will be able to access the data base. Brant noted that they intend 
to make the data accessible to as wide an audience as possible over time. 

3. David Murphey, Vermont 

Mr. Murphey noted that Vermont has had a head start on the development of indicators because 
of the leadership of its Agency of Human Services Secretary, who understands and values data. 
Vermont has developed two tracks. The first is designed to examine, at a state-wide level, 50 
indicators of social well-being. Half of these are focused on children. The indicators cover 
everything from prenatal care to youth employment. Indicators are tracked over time and against 
the U.S. average, and are ranked with the other states. Vermont does some international 
comparisons as well. Their annual report is now in its fifth year and is available on the World 
Wide Web. The second track is designed to examine indicators at a more local level. A planning 
group has identified nine outcomes for children and families. Examples include the following: 
families, youth and citizens are part of their community’s planning, decision making and 
evaluation; pregnant women and newborns thrive; and infants and children thrive. Community 
profiles on the indicators are produced at the state and county level, and at 59 school supervisory 
union levels. 

Murphey also presented a "wish list" for Vermont to support their work on indicators: 

• better automated linking of various state data systems; 

• more timely data, especially in the area of health outcomes; 

• more reliable substate population estimates for computing rates; 

• more indicators, in particular for the outcomes "Children are Ready for School," and 
"Children Succeed in School;" 

• more indicators that focus on strengths or assets of children and youth; and 
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• more and better indicators of community characteristics impacting on children. 

Session 2 State Presentations, continued 

1. Judith Erikson, Indiana 

Ms. Erikson, the Research Director for the Indiana Youth Institute, said that her organization is 
the "end user" of data. It takes what is produced by state and federal agencies and puts it into a 
form that people providing services and writing proposals can use. Her organization has worked 
hard to develop positive indicators of youth development because most existing indicators do not 
measure positive development. The Indiana Youth Institute’s "wish list" would focus on 
improving the development of indicators in this area. The Institute is also the Kids Count 
affiliate for the state of Indiana. Using the national composite rank for each of the 50 states and 
the District of Columbia, Indiana found that they were rated around 32nd in the nation. They 
have gradually worked up to around 26th in 1 996. Erikson stated that people use Kids Count 
Indicators because they go into more depth (for example, low birth weight by age of mother). 
With the county level data system available through Kids Count, they can compare various 
locales within their state. 

The Indiana Youth Institute also helped develop the Social Assets and Vulnerability Index 
(SAVI). SAVI provides visual information about a particular indicator. For example, SAVI can 
portray where welfare recipients live, where juvenile arrests occur, etc. It cannot provide 
information about individuals but can determine whether in certain Census tracts, some things 
are more or less likely to occur. SAVI allows service providers and advocates to determine how 
many boosts or community strengths it might take to raise a healthy child. A serious flaw with 
SAVI is that they do not have any positive indicators. For instance, they would like to include 
high school graduates, not just drop outs. Users of data want asset-focused as well as deficit- 
focused indicators. They also want the information in the smallest possible unit, so that they can 
aggregate it later. 

2. Jim Murkette and Ron Lucas, Connecticut 

At the present time, Connecticut is in the process of conducting an inventory of what 
administrative data relative to child well-being are available within the state. They have found 
that a lot of the information that they were looking for is already available but is difficult to 
access. Mr. Murkette noted that information systems are changing and need to be approached 
from a policy perspective. For example, policies should determine what type of data should be 
collected. The Connecticut welfare agency does not track non-marital pregnancies so they 
approached the health department and asked what type of data are collected on birth certificates. 
All of the agencies that they have approached have accommodated their needs. They have also 
truncated their computer data systems. 
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Murkette described the report that the Connecticut Commission on Children released in 1993 
called Goals and Benchmarks for 2000 and Beyond. He said, however, that the indicators that 
were included in this report were all negative. The Connecticut Association for Human Services 
and an independent nonprofit research group have established some benchmarks such as child 
poverty and low birth weight. Connecticut is also in the process of developing performance 
measures such as high school completion, child abuse, and child immunization which they may 
collect as part of the JOBS First program. They know what families are using services, and they 
have information about family formation and household structure, reports on abuse and neglect 
and other information about families in the program. Connecticut has been bringing together 
programmers and those working with data to transform their computer systems. Through the 
process of elimination, they will come up with a more delimited set of indicators and impacts. 
Mr. Lucas said that Connecticut’s "wish list" would include a data warehouse. 

3. Chuck Johnson, Minnesota 

Mr. Johnson discussed Minnesota’s welfare reform plans, how they plan to track outcomes of 
welfare reform, and how they will link this work to indicators projects. The Minnesota state 
legislature just passed a welfare reform bill, and Johnson noted that there will be an 
overwhelming amount of work over the next six months in order to implement this bill. Johnson 
said that Minnesota has been thinking about how they want to track this new legislation and how 
child outcomes fit into their plans. Minnesota is developing a data warehouse, which as 
originally conceived, was focused on producing data from individual systems, not on linking 
systems. Data systems were set up to support the operation of programs, rather than for 
reporting. Johnson noted that they are not near the point of integrating data across departments, 
though there are several exciting plans for new data systems being discussed. Development of a 
new social services system will help track child welfare information statewide. In the past, most 
child welfare data were housed at the local level. Minnesota is developing a new system for 
tracking child support that will be more user friendly. Their top priorities are to give the Federal 
government the type of data that they need, while at the same time ensuring that their programs 
are operating efficiently. Minnesota has also been dealing with how to use their Quality Control 
staff to produce useful information for tracking progress. For example, Johnson said that they 
can learn hundreds of different things about a particular family. The challenge is to build a data 
system so that counties and state level officials can receive accountability information. 

4. Pat Hall, Florida 

Ms. Hall provided an overview of what stage Florida is in with regard to implementing welfare 
reform. MDRC has recently published a report that provides results on welfare reform pilot 
studies in Florida. The results from the pilot studies have been translated into state-wide 
programs. The pilot experimental group has become the normal state-wide program. Hall said 
that her office is asked frequently for information about welfare reform. She noted that Florida is 
one of the first states in which people are reaching time limits. The dire outcomes for these 
families that many people expected have not materialized. Of 79 families which reached the 
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time limits, only three have needed out-of-home placements for their children. Also, she noted 
that they do not yet understand how people receiving assistance are using child care. Hall also 
said that Florida has not used as much of its child care funding as it anticipated it would. 

Hall noted that Florida is ahead of many of the other states in implementing welfare reform. Its 
state plan was submitted in September and the plan was implemented state-wide by October 1 , 
1996. They have done a phenomenal amount of work to accomplish this. There has been a great 
deal of change in how welfare is administered. Florida has a quasi-govemmental citizen group 
that is largely composed of business entities. They will be coordinating all state agency 
functions for welfare reform. The citizen group merges the efforts of the workforce activities. 
The data systems staff in Florida are trying to link data systems together so that they will have all 
of the information that they need about families as they move off of the welfare roles. Hall said 
that they have people "dropping off the welfare roles in droves," perhaps 5,000 and 6,000 people. 
However, Florida does not completely understand why this is happening so fast. She also noted 
that are fewer new cases because of the time limits. Hall said that they have been meeting with 
researchers at Florida State University and have shared all of the prior meeting materials with 
them. These researchers are going to be conducting a search of the literature and a review of data 
bases to find out what data agencies in Florida are already collecting. 

5. Dave Gruenenfelder and Alan Whittaker, Illinois 

Illinois has made the decision to develop some type of child indicator system. After the first two 
national level meetings, Mr. Gruenenfelder said that they brought the ideas about indicators 
back to the state and everyone liked them, though they acknowledged that this was not 
necessarily an easy decision. By producing indicators of child well-being, the state might be 
faced with its own bad news. Many of the indicators are framed negatively, and when one 
selects measures, these might be things that drive policies in the state for many years. However, 
Gruenenfelder noted that they do want to know the reasons why problems are occurring and how 
to fix them. Illinois would like to be more proactive in selecting indicators. For example, they 
should be clear and readily understandable, they should be available at the local level, and they 
should include both positive and negative aspects of child well-being. Illinois has been thinking 
about what kinds of indicators they would like to collect and how they can frame them as "goals 
for children." Examples include: children should be healthy; children should have success in 
school; and youth should be work-ready. 

Mr. Whittaker said that Illinois is about to undergo a reorganization of human services. Three 
entire agencies and parts of three others will be merged together. The governor’s office and a 
task force of state and community representatives examined issues around human services, which 
forced everyone to re-think the mission of human services. Everyone agreed that human services 
need to have a strong community focus. Community partnerships will be increasingly important 
in the future and data will need to be able to inform community-level decisions. In addition, 
there will be a switch from data about processes to data on outcomes. Services will also be 
provided in a holistic way. Those involved in the reorganization process also agreed that Illinois 
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needs a broader sense of what people need and what kind of services they are receiving. State 
agencies have been examining their own data and trying to identify useful indicators. At the 
present time, everyone realizes that the data are fragmented. Chapin Hall has done a lot of work 
in support of Illinois’ child impacts study. They have thoroughly catalogued the administrative 
data available for the state, which will be useful for the development of indicators. Chapin Hall 
is also working on linking data from various sources by conducting a probabilistic matching of 
data across multiple programs. 

Session 3 State Presentations, continued 

1. Carol Baron, Virginia 

Ms. Baron said that when she first came to the Virginia Department of Social Services (DSS), it 
was very difficult to get data. If someone at DSS wanted to do research, they would have had to 
get in line as a mainframe user. Data files were often developed for program management or 
fiscal purposes, rather than for analyses. At the present time, however, DSS collects data in 
several different areas on a regular, state-wide basis. These include: AFDC/TANF historical 
program data; AFDC/TANF historical demographic data; cumulative employment services data; 
cumulative child day care data; cumulative closed/denied/delete data; and standard random 
moment sample data. DSS also plans to add historical service program data. 

As soon as Virginia received its 1115 waiver, they identified several sources of existing data to 
use for their research sample. These data included: cumulative AFDC (TANF) payment history 
data; cumulative food stamp payment history data; cumulative VE Wage history data; 
cumulative Medicaid service history data; and cumulative child support payment history data. 
They plan to add live birth and immunization records data for their research sample. The data 
are currently on the mainframe computer and one is extracted and put on PCS on a regular basis. 
Five or six different software programs can be used to analyze the data. DSS is working on 
designing the data files so that they are useful for a broad variety of groups. In addition, people 
who are mandatory for Virginia’s work program are identified on the data file. This allows 
caseworkers to know in advance the requirements for each client. DSS is also working on 
building profiles of recipients regarding their patterns of assistance. Clients are identified by both 
a case number and a client identification number. Depending on the question for which the file is 
used, either identification number can be used in analyses. 

2. Marge Reinhart, Oregon 

Ms. Reinhart discussed Oregon’s Performance Measurement Development System. Oregon 
Shines II: Benchmarks is a state-wide effort to focus on the measurement of outcomes. These 
include child well-being indicators such as immunization, medical care, child care availability, 
and school achievement. Reinhart said that they are continuing to work on developing good 
indicators for child readiness for school. A quarterly performance update tracks the Adult and 
Family Services Division’s current performance indicators. There is a description of 
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performance measures within the agency including definitions and a sample of how the 
information is disseminated. Reinhart noted that partnerships are necessary to create 
performance measures, indicators and outcomes. Within their own agency, they do not develop 
any standards until they speak with other agencies such as JOBS, schools, health agencies, staff 
and clients. As a state, they are looking at the expansion of performance measures of child well- 
being beyond their own agency. There was a state-wide meeting which included a discussion 
about indicators, measures, inputs and outputs. People outside of the state government are also 
involved. They will soon begin the process of jointly agreeing upon measures. Reinhart stressed 
that cross-agency development has been critical because they do not always have consistent data. 
Different agencies use different definitions and different terminologies. 

Oregon is developing measures and management information tools around retention. They want 
to learn what the factors are that enable clients to stay off of assistance. They also want to find 
out what constitutes quality child care, the availability of child care and medical coverage, and 
how adults and children are involved in juvenile court proceedings. Oregon has just started to 
think about creating a data warehouse. They rely more on administrative data, rather than 
surveys, because they are more cost efficient. They recognize the imperfections in administrative 
data and are using this as a starting point to improve their data systems. Reinhart noted that 
Oregon has the same "wish list" as Vermont and the other states. 

3. Deb Bingaman, Iowa 

Ms. Bingaman said that Iowa has come a long way in their welfare reform research efforts since 
1993. They plan to conduct an evaluation and have asked the state legislature to appropriate 
funds for this work. At the present time, Iowa has the following administrative data collected on 
a quarterly basis since 1993 linked to its impact study: welfare case data; child support; 

Medicaid; wage file; and welfare individual recipient data (demographic data). They also have 
the following administrative data linked to FIP (Iowa’s welfare program) recipients: welfare case 
data; child support case data; DES wage file; and welfare recipient data. For their Limited 
Benefit Plan (LBP) study, Iowa has: Welfare case data; survey data; a LBP master file (LBP 
data, reason for assignment, Social Security number, case number); and welfare recipient data 
(demographic data) linked to their sample. 

Iowa is proposing to link survey and administrative data for its Child Outcomes Study and plans 
to have the following information for the research sample: child support; Medicaid; family and 
child services data (e.g., foster care); wages file; welfare case data; survey data; and a child abuse 
registry. Bingaman stated that Iowa has the same data "wish list" as the other states. 

4. Steve Smucker and Susan Wright, Michigan 

Mr. Smucker reported on the data sources that the Michigan Family Independence Agency has 
available to them for their research. Data that are currently available include: a client 
information system (Comprehensive Eligibility program); wage reporting; vital statistics; 
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protective services data; children’s services; Census data from prisons from the Department of 
Corrections; and Michigan state police arrests and incarceration. The Family Independence 
Agency (FIA) is currently considering several other sources of data including: a longitudinal 
data base created from the client information system; geocoding; Census data; information on 
licenses from the Secretary of State; and fishing license records from the Department of Natural 
Resources. 

With regard to their data warehouse, Michigan is hoping to have the data produced on a regular 
basis so that all information can be dynamically linked. Smucker said that the Michigan state 
police have devised a method using fishing licenses, Social Security number, and date of birth to 
create an identifier for an individual. The state police claim to have a 90% matching rate. The 
FIA provides the state police with certain information and then they return it with additional 
information about the identification of individuals. Analysts are currently developing Internet 
applications that can be tied directly into the data warehouse. This raises questions of 
confidentiality, though there is not yet a state-wide policy concerning access to data. 

The FIA has conducted several analyses with the data in order to answer some questions. For 
example, what sorts of stressors do people who are sanctioned under TANF experience? The 
FIA found that there was 50% more contact with protective services than for a random sample of 
regular, non-TANF recipients. However, when they analyzed the longitudinal data, they found 
that contact with protective services was 50% ten years ago. They concluded that families which 
are sanctioned are distressed in general. 

Ms. Wright discussed an annual report of Michigan’s Critical Health Indicators. This report 
spans several human services agencies in Michigan including the FIA, the Department of 
Education, and Community Health. The report contains 33 indicators of well-being of state 
residents. It links local and state initiatives to improve the health of its residents, and has served 
as a prototype for similar efforts at the local level. The audience ranges from the general public 
to state and local governments. The report uses lay terms, with little technical jargon. Wright 
said that it is readily understandable, compels action, and is of general concern to the public. 
There are six categories of health indicators in the first report. These include: survival; health 
risks; disease; causes of death; the health care system; and broader determinants of health. A 
clear definition of each indicator is provided in order to permit replication. Racial, gender and 
age comparisons are provided whenever possible. They also present data over time and at the 
state level. They use a three-year average for county level indicators, however, because of the 
small population in the rural counties. The second annual report will be released soon and will 
include a smaller set of indicators that are linked to strategies to improve health in the state and 
to funding streams. 

5. Comments from the Moderators for Sessions 1 through 3 

Dr. Martha Moorehouse commented that most states have been considering administrative data 
when developing indicators. However, survey data will allow states to move beyond the service 
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population. Data collected from surveys may also allow states to look at more positive indicators 
of child well-being. Service-based data are often negative because they reflect the need for 
services. 

Dr. Matt Stagner outlined several themes that were apparent across the state presentations. 

First, he said that across all the states, they are asking new questions of old data. This brings to 
mind the image of a consignment shop rather than a data warehouse. The second issue is one of 
comparisons. There are two types of comparisons— over time and across geographic areas. 

States are struggling to create historical time trends, but also to look at smaller geographical units 
within and across states. This struggle raises questions about collecting and reporting the data. 
Third, Stagner said that states have found that it is important to follow cohorts from birth in order 
to disentangle policies from other factors. States are also dealing with the question of when 
service receipt is an outcome and when it is not. It is challenging to measure outcomes, and it is 
important to think through these challenges. Administrative data systems have the measurement 
of outcomes as their goal. However, the people who need to think about goals and outcomes 
have been diverted by major issues, such as policy changes and reorganization. Stagner also 
noted that in the presentations, people did not talk much about dissemination and audiences for 
indicators work and commented that state representatives did not discuss partnerships in great 
detail. 

Session 4 Presentations from the Experts 
Moderator: Dr. Kristin Moore, Child Trends, Inc. 

1. Dr. Brett Brown, Child Trends, Inc. 

Social Indicators and Public Policy in the Age of Devolution 

Dr. Brown described a nested topology of uses for social indicators developed by Dr. Tom 
Corbett and himself. This topology identifies five uses for social indicators: Description; 
monitoring; goals-setting; accountability; and evaluation. Indicators as descriptors produce 
knowledge and can be adopted for policy-related purposes. Brown mentioned a number of 
ongoing efforts to develop better measures including the National Educational Goals Panel and 
the Behavior Risk Factors Surveillance System. Indicators can also be used for monitoring 
purposes related to social action. For example, they can be used to identify needs and plan 
programs. With regard to goals-setting, indicators can be used to track progress toward specific 
goals. The goals can be broad and general, or specific. Some examples include Healthy People 
2000 and Oregon Benchmarks. Fourth are indicators that are used for accountability purposes; 
such indicators track progress toward goals with consequences attached to them (sanctions for 
failure, rewards for success, or both). Causal links are assumed between a policy or program and 
an outcome (for example, non-marital births under TANF). 

Finally, indicators can be used for evaluation. In this use of indicators, one is attempting to 
determine scientifically which policies are effective or destructive and why. Actions and 
outcomes are linked. Brown described a specific type of evaluation, theory-driven evaluation, 
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which is designed to evaluate comprehensive community initiatives with multiple targets and 
multiple outcomes of interest. The approach one takes depends on the development of an 
explicit model. Each element of programmatic change is linked causally. Theories of change 
which underlie the model are hypothesized to be linked to other outcomes. 

Brown outlined several technical and political issues involved in developing a solid data system 
for tracking child well-being: 

• Data availability. 

He noted that some states have developed their own surveys and are exploring ways to design 
and link data systems. 

• Measurement quality. 

• Training. 

He pointed out that social indicators are tools and users need training in order to use the tools 
responsibly. 

• Politics. 

Challenges increase when going from the least to the most politically sensitive uses of social 
indicators. 

2. Bill O’Hare, The Annie E. Casey Foundation 
Lessons Learned from Kids Count 

Mr. O’Hare first described the Assessing the New Federalism project being conducted by The 
Urban Institute and Child Trends, Inc. There are three major data components to this project: 

• An intensive survey of low-income families with children in 1 3 states: Massachusetts, 
New Jersey, New York, Alabama, Florida, Mississippi, Wisconsin, Michigan, Texas, 
Colorado, California, Minnesota, and Washington. 

• A 50-state database which is available on The Urban Institute’s Web site 
(http://www.urban.org). Data are available in a comparable format across all states. 

• In-depth case studies of state policies and practices. 

O’Hare then provided an overview of the Kids Count effort that the Annie E. Casey Foundation 
sponsors. The first national Kids Count report was published in 1990 and was an attempt to raise 
the public’s awareness about the well-being of children. The Annie E. Casey Foundation 
provides four-year $400,000 grants to the state-level Kids Count programs and organizations. 

The Foundation has spent more than $30 million over the last seven years on Kids Count, which 
does not reflect the total cost of the project since many state Kids Count projects also receive 
money from other sources. Often, partnerships are created across several organizations and each 
Kids Count project often involves more than one partner. 
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Below are observations about the Kids Count project that O’ Hare outlined: 

• There are a lot of data available at the national level that are not available at the state 
level. Similarly, there are data available at the state level that are not available at the 
county level. 

• As one goes from the national, to the state, to the substate (e.g., county) level, the burden 
of information shifts from surveys to administrative data. This has several implications. 
Data may be available from Federal surveys, but it cannot be available at the state-level. 
This leads to a reliance on administrative data which generally tends to be focused on 
some sort of problem. 

• With most administrative data bases, one will only have information on the children in 
the system. For example, if welfare caseloads go up, states will not know whether there 
are more needy children, or if more needy children are getting help from the system. 

• There are two different models by which groups attack the problem of reporting on child 
well-being. The first model involves developing a "laundry list" or "wish list" of what is 
important to measure by involving all sorts of people from multiple organizations. In 
reality, only a few of these items are actually measured. The second model involves 
finding out what data are available and then determining what one wants to learn from 
that data. The second model is the one that was more successful for Kids Count grantees. 

• A sustained effort has been critical to the success of Kids Count. The first Kids Count 
report was planned as a "one shot" wall chart. Now, Kids Count is published on tin 
annual basis. 

• There have been a lot of technological and logistical challenges in producing Kids Count 
reports. Some of these challenges have been easier to handle than others. Getting diverse 
organizations to work with each other can be difficult. Developing public-private 
partnerships with agencies outside of the government is also challenging. 

• It is important to develop a communications and dissemination plan. Once the report is 
compiled, people need to determine how it will be disseminated and how it might affect 
programs. A general rule of thumb that O’ Hare uses for grantees is that one-third of the 
grant should be used for data collection, one-third should be used for analysis and 
writing, and one-third should be used for dissemination. 

• In order to collect the best data, a team composed of representatives from a variety of 
organizations should be developed. 

• Once an organization produces credible data, they often have credibility regarding 
policies and programs. However, these things are not necessarily connected to each 
other. 

• Focusing on data and indicators is a very useful way to bring groups together. The Kids 
Count Grantees have brought people together from a variety of perspectives, such as 
health, education and the business community. 
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3. Christopher Botsko, Child Trends, Inc. 

Some Federal Data Resources 

Mr. Botsko first described a report that Child Trends, Inc. produced for the Annie E. Casey 
Foundation. The report describes several federal data systems from which state-level estimates 
can be produced. Botsko noted that with regard to survey data, it is difficult to get good state 
estimates because of financial constraints. The National Survey of America’s Families, a 
privately funded survey, is expensive, complex, and very staff and time intensive to conduct. 

Botsko described several Federal data sources by topical area that are available to states and that 
can produce good state or local measures of child well-being: 

• Population 

Decennial Census 
Current Population Survey 

• Health 

Vital Statistics system 
National Immunization Survey 
Behavior Risk Factor Surveys 
STD Surveillance System 

• Education 
Common Core of Data 
Schools and Staffing Survey 

• Crime 

Uniform Crime Reports 

Botsko also noted that there are Federal data sources available for some states: 

• Population 

American Housing Survey 

• Health 

Youth Risk Behavior Survey 

Pregnancy Risk Assessment Monitoring System 

National Health Interview Survey 

• Education 

National Assessment of Educational Progress 

• Crime 

National Crime Victimization Survey 

Botsko presented poverty estimates as an example of how a Federal government organization is 
using national level data to produce state and county level estimates. The Bureau of the Census 
is using the Current Population Survey and data from income tax returns to create poverty 
estimates at the state and county level. Botsko noted that the 1993 figures are a work in progress 
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and the Census will be producing new estimates every two years. Congress has also requested 
that the Census produce poverty estimates at the school district level, but the Census Bureau 
currently doubts that it can provide good estimates because of an inability to match tax records to 
school districts. 

Botsko also presented information about the American Community Survey (ACS). The goals of 
the ACS are: 

• To update the Census long form profile of communities throughout the decade. 

• To provide annual data to help in evaluating programs affecting local communities. 

• To replace the 2010 Census long form. 

• To provide benefits for other Federal statistical programs. 

There will be yearly ACS estimates for all places or groups of 65,000 persons. For all places or 
groups of 50,000-250,000 persons there will be estimates every two years. For places or groups 
less than 15,000 including small towns, townships, block groups or Census tracts, estimates will 
be averaged over five years. The number of years that will be averaged varies from 2 to 4 years 
for places and groups between 15,000 and 65,000. Once the ACS is up and running for 5 years, 
updates will be produced annually for all places and groups by revising the estimates by 
including data from the most recent year. 

The estimates and the methodology for developing them are constantly changing. This summary 
is a revision to the talk that was given and reflects the latest information that was available in 
June of 1997. The ACS is subject to shifting budget priorities, but barring deep budget cuts it 
should be a good resource for indicators projects and policy planning. Botsko said that the ACS 
does have some limits for the study of child well-being. For instance, there are no data on 
children’s health or emotional adjustment. 

4. Dr. Susan Hauan, Office of the Assistant Secretary for Planning & Evaluation, 

U.S. DHHS 

SLAITS: New Opportunities for getting state-based estimates in a federal survey 

Dr. Hauan said that the primary goal of the State and Local Area Integrated Telephone Survey 
(SLAITS) is to develop a national capacity to generate high quality data at the state level that 
will meet the needs of DHHS and the states. SLAITS is a survey, but really more of a 
"mechanism" by which several surveys are combined together. A questionnaire for a pilot study 
which contains a pool of potential questions has been designed. These include: questions from 
the 1996 redesigned National Health Interview Survey (NHIS) (family, adult and child core 
questions); questions from the Survey of Income and Program Participation on child well-being 
and child care; child care questions from the National Household Education Survey; child well- 
being questions from the National Survey of America’s Families; employment and program 
questions from the Current Population Survey and the Survey of Program Dynamics; and 
standard sociodemographic questions including age, sex, race/ethnicity, education, and family 
income. 
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The first module of the survey, from the NHIS, focuses on health insurance coverage, health 
status, access to care, utilization of services, and program participation questions. The second 
module is comprised of the child outcomes questions (see Table 1 in Appendix for overview). 
Intervening mechanisms that are measured in the survey are provided in Table 2 in the Appendix. 

A feasibility study is being conducted in four to five sites where they will be testing sampling 
mechanisms, the reliability of state estimates and adjustments for non-telephone households. In 
May, the pilot study will field just the health module in two states (Iowa and Washington). In 
the fall, two or three more sites will be added. The welfare module with the extended welfare 
and child well-being questions will be fielded in at least two of these sites. Dr. Hauan mentioned 
that they plan to do some sort of oversampling in order to have more children, particularly low- 
income children, in the sample. 

The initial wave of the study is federally funded, but ASPE views future study efforts as a 
partnership with interested states. After the pilot work is completed, they will need to develop a 
cost-sharing mechanism that will vary depending on customization and over-sampling needs. 

Dr. Hauan said that the tentative plan is that the Federal government will provide start-up costs, 
partial funding to allow comparisons, and technical assistance. The cost for the partnership is 
estimated to be between $50,000 and $70,000. 

5. Dr. Robert Goerge, Chapin Hall Center for Children, University of Chicago 
Working with Administrative Data 

Dr. Goerge described Chapin Hall’s role in reviewing and organizing the available 
administrative data bases in the state of Illinois. As part of the reorganization of Illinois’ data 
systems, Chapin Hall considered the entire population of individuals who would be served by a 
variety of agencies (including Medicaid, child welfare, Food Stamps, and AFDC) as if they were 
served by one agency. A total of 1,871,000 people, representing 16% of all Illinois residents, 
received services. In addition, Chapin Hall examined the data by age of recipient, and learned 
that the Departments of Public Aid, Public Health and Child and Family Services primarily serve 
young children. Children ages 0 to 17 constitute 50% of all people who have contact with the 
system, representing approximately 30% of all children in Illinois. 

Over a one and a half month period, Chapin Hall linked administrative records for over 4.5 
million people over a five-period (from 1991 to 1996). Goerge said that the idea that one cannot 
examine the entire population with administrative data are incorrect. He noted that 
administrative data can be used in several ways. These include: 

• Using multiple data bases including tax and birth data to examine a broad population. 

• Conducting an administrative census in order to get information on the population. 

• Using administrative data to improve estimates of various population indicators. 

Goerge stressed that linkage is important not just across data bases, but also within data sets. He 
also said that there is no such thing as a "unique" identifier which can be used to link an 
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individual’s records from multiple data bases. Finally, Goerge said that in order to create a data 
warehouse, one needs data processors with an awareness of how the data is being used. 

6. Con Hogan, Agency of Human Services, Vermont Forging Ahead 

Mr. Hogan began his presentation by commenting that he spent some time in the business 
world, where he realized that indicators are the fundamental currency of progress. In order to 
know whether child well-being indicators are getting better, one needs to collect historical trend 
data. Hogan also noted that government agencies at all levels are known for constructing non- 
comparable data sets. It is important for Vermont to compare itself to other states and countries 
to know how it is doing on certain indicators. For example, Vermont’s teen birth rate compares 
favorably to the United States as a whole. However, Vermont’s teen birth rate does not seem 
that low when it is compared to Switzerland’s. 

Hogan called indicators "mirrors" to show how a local area is doing. He described the city of 
Barre which had child abuse rates that were three times those of the county and state. The city 
also had unmanageable delinquency and teen pregnancy rates. When the city was given these 
data, they were not happy with how things were going. They are selecting two or three 
indicators and developing a plan to change them. Hogan said that indicators can become 
interactive. They work together, in that change in one indicator leads to change in others. 

When people are given solid, positive goals, agencies become mobilized. Indicator progress 
gives the public confidence that the government knows what it is doing. Positive goals are things 
that the public can relate to. Hogan said that indicators open the door to the business community. 
Good indicators are assets, whereas negative indicators are liabilities. Five years of developing 
indicators in Vermont has started to pay off. The indicators are working in the right direction. 

For example, the state government is collaborating with local governments that are motivated to 
take action because of indicators. (The full text of Mr. Hogan’s prepared remarks is included on 
page 27 in the Appendix.) 

7. Linda Kohl, Minnesota Planning Spreading the Word: Dissemination of Indicators 

Ms. Kohl described the Minnesota Milestones reports produced by Minnesota Planning. The 
report development process began by asking approximately 10,000 people from a wide variety of 
departments and groups what they want the state to look like in 10, 20 and 30 years- 4 and how they 
would want to measure these goals. They were not as concerned with comparing Minnesota with 
the rest of the United States, but rather to measure Minnesota’s progress on certain indicators. 

The first Minnesota Milestones report was published in 1 992, and two updates have been 
produced, one in 1994 and one in 1996. There are full and abbreviated versions of each report. 
They have developed 20 broad goals and 79 indicators. Kohl said that they tried to produce 
indicators for several areas of interest, but learned that the data were not available. They ended 
up producing their own Crime Victimization Survey to answer some of their questions. 
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Another product that Minnesota Planning has created is the Children’s Services Report Card. 

The Report Card is an on-line database that is available on the World Wide Web. The user can 
click on a specific county within Minnesota and obtain information on 21 indicators, including 
indicators of child well-being. 

Minnesota Planning has made extensive efforts to disseminate the Minnesota Milestones reports 
and the results have been tremendous and positive. For example, newspaper reporters have 
started to ask county directors questions about what is going on in their county. 

Kohl said that at the present time, the Minnesota Milestones goals are not linked to specific 
programs, nor are they linked to funding decisions. They have begun to think about how some of 
the goals can be linked to programs and how they can be developed to use for funding decisions. 

8. Dr. Tom Corbett, Institute for Research on Poverty, University of Wisconsin 
Using Indicators to Evaluate Welfare Reform: Results from a Discussion of State Welfare 
Administrators 

Dr. Corbett described devolution and the emerging policy world as a period of great volatility. 
He noted that states will likely see two effects. The first, a price effect, will mean that the states 
will have to pay for the full cost of devolution "at the margin." Thus, they will need to make 
wise choices about allocating their funds. The second effect is one of technology. Corbett said 
that the locus of control will move from Washington and the state capitals down to lower levels 
of government. As the decision making moves down to the county and even subcounty level, 
better performance indicators will become even more important in order to maintain some level 
of accountability. Corbett stressed that it is important to develop critical indicators that can be 
measured using acceptable techniques. Indicators become increasingly important because it is 
harder to conduct an "old style" evaluation through experimental designs. 

Corbett described the Welfare Peer Assistance Network or WELPAN, one of several strategies 
that have been proposed to respond to the new federalism (see Table 3 in Appendix). WELPAN 
is a consortium of welfare officials from seven Midwestern states that have held regular meetings 
to discuss the new policy world. The participants decided that the first two issues that they 
needed to address were defining success and changing the culture of the welfare agency. 
WELPAN members discussed important domains and constructs to measure in order to assess 
the effects of welfare reform. They created a list of primary and secondary domains and decided 
which domains were more and less important. A draft document outlining these domains has 
been produced and serves as a consensus document on where these states are heading. 

WELPAN members also discussed the attributes of the emerging policy world, as compared to 
the old policy world. These appear in Table 4 in the Appendix. Corbett said that the specific 
choices that the WELPAN members have made may be less important than the dialogue that 
produced these choices. There are things that states will not do in isolation, but they may be 
willing to consider them when they work together with colleagues in other states. 
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Corbett also said that it is one thing to warehouse data and select indicators, but it is another 
thing to use indicators wisely. Data can be misused and it will take a lot of discussion before 
everyone can effectively and judiciously use what is an emerging technology. 

Concluding Remarks 

Dr. Howard Rolston, ACF, U.S. DHHS and Ann Segal, ASPE, U.S. DHHS 

Dr. Rolston closed the meeting by stressing the importance of the work that the states are 
conducting. He also said that ACF would try to find a way to fund these indicators studies, in 
addition to the child impacts studies. Ms. Segal said that ASPE will continue to support 
indicators efforts in the states. She mentioned an upcoming indicators conference in June at the 
Joint Poverty Center at Northwestern University, one of the efforts that ASPE is supporting. 
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Inyminnegandia Scenario 1 




Non-marital -^Poverty -^Foster Care Welfare Caseloads 

birth rate (percent) (per 1000) 



Inyminnegandia Scenario 2 




N on-marital ~ Poverty Foster Care -m- Welfare Caseloads 

birth rate (percent) (per 1000) 



Inyminnegandia Scenario 3 




^ Non -marital Poverty -♦ Foster Care m Welfare Caseloads 

birth rate (percent) (per 1000) 
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Table Is STATE AND LOCAL AREA INTEGRATED TELEPHONE SURVEY (SLAITS) 

CHILD OUTCOME CONSTRUCT AREAS 



Health/Physical 

Development 


Cognitive/ 

Achievement 


Social/ Emotional 
Development 


Physical 

Environment 


General Health Status 
(NHIS) 


School Enrollment 
(SIPP) 


Mental Health 
(NHIS) 


Neighborhood 
environment/ 
safety (SIPP) 


Height and Weight 
(SIPP) 


Highest Grade 
Completed 
(NHIS, SIPP) 


Sports Teams 
(SIPP) 






Gifted Student 

Programs 

(SIPP) 


Lessons (Outside of 
School) 

(SIPP) 






Repeated Any Grades 
or Held Back 
(SIPP) 


Clubs or 

Organizations After 

School 

(SIPP) 








Suspended or 

Expelled 

(SIPP) 








School Contacted 
Parent about Problem 
(NHES) 





Key to Survey Acronyms: 

SIPP: Survey of Income and Program Participation 

NHIS: National Health Interview Survey 

NFS: . New Federalism Survey 
NHES: National Household Education Survey 
FCS: Food and Consumer Services 
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Mother’s 

Psychological 

Well-being 


Mental Health/ 

Depression 

(NHIS) 








Stress 


Parental 
Aggravation 
Measure (NFS) 








Responsible 

Parenting 


Visits to 
Library, 

Family Outings 
(NHES, SIPP) 


Rules about 
TV 

(SIPP) 


Reading to 

Children 

(SIPP) 




Child Care 


Type of Child 

Care 

(SIPP) 


Cost of Care 
(SIPP) 


Amount of 
Time in Care 
(SIPP) 


Self Care 
(SIPP) 


Health Care 
Access 


Access to 
Medical Care 
(NHIS) 


Health 

Insurance 

Coverage 

(NHIS) 






Stability/ 

Turbulence 


Changes in 
Child Care 
(NHES, SIPP) 


Changes in 
Living 

Arrangements 
(SIPP, NFS) 


Changes in 

School 

(SIPP) 




Employment 


Employment 
Status (SIPP) 


Hours and 
Weeks of 
Employment 
(SIPP) 


Shift Work/ 
Regular Hours 
(SIPP) 


When Last 
Worked (NFS) 


Income/ 

Economic 

Stability 


Income 
(SIPP, NHIS) 


Program 
Participation - 
AFDC/TANF, 
SSI, Medicaid, 
Food Stamps 
(SIPP, NFS, 
SPD) 


Food 

Insecurity 

(FCS) 
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THE IMPORTANCE OF INDICATORS AND WHAT THEY CAN DO 
Presentation by Cornelius D. Hogan, Secretary, 

Vermont Agency of Human Services 
April 28, 1997 

Tracking indicators and outcomes should be at the center of all of our work, and yet, it is one of 
the things, that government at all levels, has not done well. And I believe because we have not 
done that work well, we have handed the crucial agenda of the well being of our people to others. 
At this crucial time, we need to be at the center of the agenda of improving the well-being of 
children and families. 

First, I need to make the comment that I spent eleven years in a tough business environment, 
including a successful Chapter 1 1 experience. And if you ever wanted to understand the 
importance of indicators, in that environment, without indicators that adequately describe the 
direction of the enterprise, you will not succeed. And those years outside of government drove 
me to make indicators the centerpiece of the work that we do at Human Services and beyond in 
Vermont. 

Let's review for a minute some of the fundamental reasons why using indicators to guide our 
work is important if not essential. 

They tell vou where you've been, where vou are, and can guide you to where 
you need to be 

It is so elementary simple a concept that we often forget how to apply this kind of thinking. 
Taking stock is a normal and important part of the human enterprise whether it be family, work, 
or community. And knowing the trend line, are we getting better or worse over time, is one of the 
most fundamentally straightforward questions that we all need to be able to answer. It is the 
ultimate accountability. I am taken back to one of the political speeches President Reagan made 
during the election campaign against Carter when he asked the basic question, "Are we better off 
or worse off today than we were four years ago." The fundamental response of the public to that 
basic question was astounding, and in many ways shows the power of that kind of thinking. 

Indicators can help us understand how we are doing compared to others 

Knowing how we compare to others is an important part of the indicator process. Government 
agencies and organizations are famous for creating non-comparable data. One of the criteria that 
I try to use is to use data that can be compared, not only to other states, either individually or as a 
nation or region, but in certain cases comparable to international data. For example, recently we 
are seeing some decline in teen birth rates in Vermont and I can show that same data for the 
nation, where we seem to be moving in the same direction. However, it is an important, sobering 
and humbling fact to know that teen pregnancy rates are still seven times that of Switzerland. So 
the ability to compare is so very important. 
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Over time indicators give vou the basis for cost benefit analysis 



Now that may sound like a technical achievement, but it is a very important political one. In my 
state, we have been making the proposition to the business community and the Governor's 
Economic Advisors, and others who understand the concept of investment and apply it daily in 
their own work, that by tracking indicators over time, we can make some solid assumptions 
about cost structure. And it is a particularly good story when indicators are going in the right 
direction. That is the environment in which we can talk about multi-million dollar less demand as 
a result of performance over the last five years. For example, we can now show that the reduction 
in child abuse from unacceptably high levels some five years ago, results in many millions of 
dollars of demand on the system which we do not have to deal with, thereby giving us more 
opportunity to put our increasingly scarce dollars into more thoughtful child and family friendly 
investments, rather than continuing the inexorable climb of the cost of back end programs. 

Indicators can become self-fulfilling 

I want to give you an example of the power of holding a mirror up to a community and whereby 
the holding up of the indicators becomes a process where a community wants to play a major 
role in changing those indicators. We just published our second year of localized indicators and 
we have one city, the City of Barre, where some important indicators were frankly not good. In 
an environment where the state's child abuse numbers have gone down 30% in the last four 
years, Barre's was two times that of the county average within which it resides, and had teen 
pregnancy rates at the three times level. Even in the context of other positive indicators the report 
caused concern. 

That report resulted in an open community debate, including not only service providers, but 
elected officials and business people, and has resulted in the beginning of a three year process 
with the objective of changing some key indicators. The indicators have brought a lot of people 
together, and when they looked in the mirror, they didn't like what they saw, and they are going 
to do something about it. There is no doubt in my mind that the self-fulfilling aspect of using 
indicators to bring about change is working in many places across our little state. 

There is a principle in physics known as the Heisenberg principle, where the yard stick you use 
to measure the results, actually changes the results. The Barre project is a classic example. 

Indicators become interactive and help build critical mass for change 

There was a wonderful piece in the New Yorker about a year or so ago called "The Tipping 
Point." In the article, they examined the literature regarding the dynamics of change and used an 
epidemiological view applied to social science. In essence, the article points to the inter-activity 
of factors that can quickly add to a tipping point, either positive or negative in the unfolding of 
the dynamics of a particular epidemic. 
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In Vermont, for example, after seeing a few of our input indicators improve dramatically over the 
last few years, such as the percent of population covered by health insurance, the percent of 
women with early pre-natal care, the percent of newborns receiving home visits, we've seen 
outputs interactively flow. For example, we've seen teen pregnancy rates, child abuse, and 
numbers of children needing special education decline. 

Well constructed indicators cross agency lines thereby setting a common target for all of 
our efforts 



Across organizations, indicators are the common language, and the common currency. They 
bring common purpose to our multi-faceted work. For example, children ready for school 
embraces virtually the work of all of our organizations whether they be inter-generational reading 
programs, or health programs for children. And they bring together disparate organizations, 
regardless of their hierarchal ranking, to all contribute, in one way or another to the common 
outcome. 

Indicators brine the public confidence that we know what we are doing 



This is something that is very important, particularly in the era of the continuing public 
dissatisfaction and actual distrust of all things government. Indicators at a common sense level 
are a different way into the public's consciousness about government. It is not the normal kind of 
bureaucratic face. My indicator report in Vermont uses one indicator per page. The media 
understands how to use that simplicity. The Agency probably had fifteen stories in the last three 
months that flowed from this very straightforward approach. Providing health insurance, 
reducing teen pregnancies, reducing child abuse, increasing parentage rates, are language and 
body English that the public can understand. When the public understands what you do, you are 
much more likely to get support for what you do. 

In essence, outcomes and indicators convey critical and complex information in a way that is 
easy for most people to readily understand... and act on. 

Using indicators washes a lot of the politics out of budgetary and short term decision making. 
The indicators are what they are, and tend to be much more dispassionate and objective than the 
normal here and now and politicized way of presenting information that leads to budgetary and 
political food fights. 

Well constructed indicators over time systematically build budgetary and 
political support 



In our little state, this is the fifth year that we have produced statewide indicators and the second 
year that we have produced localized indicators. In this short time, this work has resulted in a 
change in our legislature, whereby the concept of investment has been incrementally and more 
readily accepted, resulting in a continuous and modest stream of investments ranging from the 
prevention investments in child care, transportation, education and training, and welfare reform, 
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to continuous modest appropriations for Success by Six, which now become targeted at particular 
areas of the state where we need to improve specific indicators. 

Indicators open the door to the business community 

It is simply a fact of life that the business community has a lot to do with our success or failure in 
generating resources, and support for the work that we do, and I make that case particularly at the 
local level, where state budgets are scrutinized and commented upon by the business community 
in very strong fashion. One thing that human services and social services has not done well is 
relating to the business community, particularly in the qualities of our language. 

Outcomes and indicators allow a direct bridge into the mind of the business community. Having 
come out of a rather difficult business situation myself, I can tell you that indicators are at the 
center of the business process. 

Business balance sheets are the life blood of business decision making. I propose to you that 
formatting indicators in a business-like balance sheet orientation, can generate a lot of invisible 
and quiet support for your efforts. 

If you can envision the classical balance sheet, which, on the asset side would show current 
assets, physical assets, and intangible assets, and then envision a typical liability side of a 
balance sheet which would show short term liabilities, long term liabilities, and other liabilities, 
all of which add up to a business equity statement which speaks to issues of ownership, profit, 
retained earnings, and overall equity, I would submit to you that there is a terrific parallel that 
can be developed for our indicator world. 

Please envision current assets as being some of the foundation assets that are leading indicators 
such as percentage of population covered by health care, and percentage of population receiving 
early pre-natal care. Other assets would include those positive indicators that the driving assets 
produce, such as reductions in child abuse, reductions in teen pregnancy, increases in parentage 
rates, and decline in hospitalization rates. You get the idea. 

On the liability side of the ledger, in terms of current liabilities, would be indicators that are 
beginning to turn negative in their early stages. Longer term liabilities would be those chronic 
indicators that are negative in nature which depress the capacity to produce better results. High 
poverty rates, poor general economic conditions, are examples of long term liabilities. 

The equity of the enterprise overall would be translated into the overall condition of the people 
with more specific consideration as to whether that condition is changing over time. 

One neat twist is the consideration of intangible assets, which I am beginning to equate with the 
energy and capacity that a local community and neighborhood has to take on more responsibility 
and authority as it relates to actions and efforts to either improve the assets indicators, or 
minimize the liability indicators, in other words, improve the condition of the people. 
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Now in short form, I've simply converted a lot of the work that you and I do on a regular basis 
into language that business people not only understand, but support. I would submit to you that 
this systematic conversion, in front of Rotary clubs, Business Roundtables, and other business 
interests, will go a long way for business understanding that investments in the work that we do 
will have payoff in terms of social capital. On a business profit and loss statement, those 
investments can easily be viewed as the cost of advertising. Investments in public health 
education, for example, can easily be translated into the cost of advertising, and the investment 
needed to return profit. I've spent a little more time on this particular aspect of indicators than I 
normally would, because I think it is such an under-developed area with tremendous 
opportunities for fundamental connections with such an important group. 

Summary 

So the summary of these thoughts is that using outcomes and indicators around which to 
organize and present our thinking, is just not something nice or a current fad. Outcomes and 
indicators are fundamental to the success or lack of success of our work and lives. Organizing 
our thinking and our systems in this way, over time, will result in a better condition of our 
people, result in less back end cost and improve the argument for more investments in 
prevention. 

And, I want to compliment Health and Human Services (HHS) on its current indicator work as it 
relates to the (Senator Daniel Patrick) Moynihan '94 budget welfare indicator requirements. I 
hope that work is providing some guidance for you. 

Finally, I compliment HHS for the Child Trends work because it is made to order for bringing 
outcomes and indicators to the center of the welfare reform debate. The only moral and 
acceptable standard that justifies welfare reform, is improving the condition of children and 
families, which can be measured, and must be a centerpiece of the overall welfare reform 
process. And in that regard, you folks are leading this overall outcome and indicator process at a 
time when we so badly need it. 
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Table 3: EMERGING STRATEGIES FOR THE NEW POLICY WORLD 



Strategy Definition 



The Guild Strategy 

The Information/Technology Diffusion 
Dissemination Strategy 

The Expert Review Panel Strategy 

The State Peer-Assistance Network Strategy 

The Technical Assistance Strategy 



Work toward common research questions, 
methods, terms and definitions, indicators and 
outcomes of interest, and reporting of data. 

Inform and influence a variety of stakeholders 
including: State officials, evaluation firms, 
academics, foundations and federal officials. 

Identify credible results and new and emerging 
issues, and cross-walk results. 

Replicate the use of regional peer assistance 
networks of welfare officials representing 
contiguous states. 

Direct help to states to improve evaluations. 



Table 4: ATTRIBUTES OF THE POLICY WORLD: EMERGING VS. OLD 



Emerging Attributes 


Old Attributes 


Behavior Oriented 


Benefits Oriented 


Dynamic 


Static 


• Point-in-process 


• Point-in-time 


• Longitudinal 


• Monthly 


Complexity, Volatility, Variability 


Simple, Stable, Same 


Multidimensional/Individualized 


Single-Tracked 

Equity/Equality 


Multiple Goals/Objectives 


Limited Goals 


Multiple Targets of Interest 


Limited Targets 


Flat Organizational Structure 


Hierarchical 


• Horizontal Communications 


• Vertical 


Professional Model/Discretionary Decision 
Making 


Nonprofessional/Controlled 


• Craft Orientation 


• Bureaucratic 


Entrepreneurial 


Conformance 


Client Responsibility 


Agency Initiative 


Team Worker 


Independent Worker 


Community Agency 


Autonomous Agency 
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